Event source management using a metadata-driven framework

ABSTRACT

The system and method described herein relates to managing multiple network device connections, collecting event source data from one or more network devices with one or more collectors, filtering the event source data, continuously monitoring the network device connections, controlling raw data collection from the one or more network devices, parsing the event source data into normalized data structures, and managing configurations for the collectors, among other things. Event sources may be physical or logical network devices distributed across a network, including but not limited to, firewalls, routers, biometric devices, mainframes, databases, and applications. The network devices may be sources from which the collectors may receive or request the event source data.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.13/159,927, entitled “Event Source Management Using a Metadata-DrivenFramework,” filed Jun. 14, 2011, now allowed, which is a divisional ofU.S. patent application Ser. No. 11/730,493, entitled “Event SourceManagement Using a Metadata-Driven Framework,” filed Apr. 2, 2007, nowU.S. Pat. No. 7,984,452, which claims the benefit of priority to U.S.Provisional Patent Application Ser. No. 60/865,305, entitled “EventSource Management Using a Metadata-Driven Framework,” filed Nov. 10,2006, the contents of which are hereby incorporated by reference intheir entirety.

FIELD OF INVENTION

The present invention relates to monitoring and managing computernetwork event sources and particularly to providing network device levelvisibility for event source management.

BACKGROUND

Currently, computer network management systems require many resources tomanage and monitor multiple network device connections. Present managingsystems can visualize connections to network devices, however thesesystems fail to visualize and manage operations at the actual enddevices. As such, these systems are limited in their ability to manageand collect event sources at a network device level. Furthermore,current network management systems do not provide any insight into thestate of the event source itself, the network device. This prevents thesystem from taking actions on the network device and can results in asystem that is limited to reacting to event data rather than proactivelypreventing errors, and/or security breaches using the state informationfrom the network device. Current systems also lack, among other things,the ability to normalize event data from disparate event sources forefficient reporting and analysis.

SUMMARY

The present invention overcomes these and other disadvantages. Oneaspect of the invention relates to managing multiple network deviceconnections and/or collectors for collecting event source data from oneor more network devices, filtering event source data, continuouslymonitoring connection status to the one or more network devices,controlling raw data collection from the one or more network devices,parsing event source data into normalized data structures, and/ormanaging collector configurations, among other things. Event sources maybe network devices (physical or logical) across a network, including butnot limited to, firewalls, routers, biometric devices, mainframes,databases and/or applications. A network device may be a source fromwhich a collector may receive and/or request event data.

The present invention provides a flexible collector framework including,among other things, one or more collectors that gather events andinformation from numerous network devices, systems and applications. Thecollector framework also provides one or more user interfaces that allowusers (e.g., network administrator) to manage, control, and monitorevent sources (e.g., network device(s)) and connections to the eventsources. Data from the event sources may be collected and normalized inorder to more efficiently determine the level of security and complianceto current network standards and/or network governance for corporate (orother) network having disparate devices.

A system and method of the invention provides network device levelvisibility which allows data from network devices to be monitored,filtered, managed, and/or configured in a proactive manner. Furthermore,for effective management, multiple network devices can be pooled to asingle collector.

The collector framework may comprise a plurality of components which auser can deploy, configure, and monitor via one or more user interfaces.The components of the framework system may include a central manager, acentral repository, a proxy manager, a collector manager, a collectordirectory, an event listener, an agent engine, one or more collectors,and/or one or more network devices. One or more of the network devicesmay be grouped into network device groups. Collectively the one or morecomponents in the collector framework enable the one or more collectorsto be deployed in order to perform in-depth monitoring into events andstatus (state) related to one or more network devices. Status mayinclude operations state information for the device (e.g., normal,errors, unsecured, etc.). Events may include, among other things,security events, compliance events, or network traffic events.

The central manager may function as a control unit. A central repositorymay be located at or associated with the central manager. Event data andstatus information may be received and collected at the centralrepository. The central manager also provides user interface viewsincluding a collector configuration interface, a health/statusmonitoring interface, and/or a central repository interface. Eachinterface enables a user and/or network administrator to visualize andcontrol respective aspects of the collector framework. For example, theuser may configure a collector using the collector configurationinterface. The configuration information (e.g., collector scripts,configuration settings, test data, etc.) may then be propagated from thecentral manager out to a collector manager.

The collector manager may execute several functions including, amongother things, connecting to an agent engine, creating a collectorprocess on the agent engine, managing listener ports for event data fromthe agent engine, processing event data from the agent engine, sendingprocessed event data back to the central repository, and/or sendingstatus of the running collector(s) to a proxy manager. The collectormanager may start a collector at the agent engine by downloading thelatest version of a collector script from the central repository and/orreceiving a collector data object. The downloaded script may be writtento the agent engine to begin a collector process. The collector dataobject may be retained at the collection manager in order to maintain acopy of one or more collector instances.

The proxy manager may execute several functions including, among otherthings, managing connections to network devices, which are the source ofdata for a collector, updating collector related configurations,triggering the collector manger to start a port and passing it therelated collector data object, and sending status (health) ofcollector(s) to the central repository. Communications with the proxymanager are carried out in proxy mode.

The agent engine may operate as a security device for event exchange.The agent engine may instantiate a collector based at least in part ondata obtained from the proxy manager, collector manager, and/or centralmanager. The agent engine receives and manages the event data from thecollector and sends the event data to the collector manager and/orstatus information about the collector(s) to the proxy manager.

The collector may be linked to one or more network devices and mayinclude a port configuration module and a collector script.Collectively, the port configuration module and collector script createa collector. Each may be configured independently of each other by thecentral manager or other entity. A unique port configuration file iswritten for each collector. A port configuration file contains theconfiguration information for opening one or more communication ports torespective network devices. The collector may be deployed in order toperform in depth monitoring of the state of the network devices. Thecollector manager may download the collector scripts from the centralrepository in order to run a collector. Collector scripts may be devicespecific. The collector script parses the data from the network devicesinto a normalized data structure. This allows the system to effectivelynormalize event data from disparate network devices for easierstandardized reporting of event data.

One or more common network devices may be grouped to create a networkdevice group that feeds event data to a single collector. Networkdevices may include, among other things, firewalls, routers, biometricdevices, mainframes, databases and applications. Event data may includesecurity events, compliance events, network events, etc.

The event listener is a socket server on the collector manager thataccepts connections from agent engine. The agent engine sends event datato the event listener for processing by the collector manager.

A proxy connector is a component that connects the agent engine to theproxy manager. The proxy connector enables data and messages to becommunicated back and forth between the agent engine and the proxymanager during a proxy mode. The proxy connection may be used tocommunicate status/health information from the collector and devices tothe central repository.

According to one embodiment, the collector framework enables a user toaccess the collector configuration interface to create and/or update acollector from the central manager. Instructions may be propagated outto the collector manager that may in turn be used to create and/orupdate the collector(s).

According to another embodiment of the invention, the collectorframework allows device level status (health) monitoring. A healthmonitoring interface can give visibility into the operations status ofthe network device. The collector script associated with each collectormay gather status information from the one or more respective networkdevice. The data may be normalized to provide standardized statusreporting.

According to another embodiment of the invention, the system providesthe ability to test event source connections. This functionality canallow the user to test configuration settings by activating a connectionto an event source (e.g., network device) and displaying raw eventsource data to the user. The system may also enable the user to select acollector or event source and automatically display those events comingfrom the selected event source (e.g., network device). Thisfunctionality may also allow a user to view raw event source datastreaming in real-time from the selected event source.

According to yet another embodiment of the invention, event sourcefilters may be configured at the network device level in order toimplement data filtering at the network device and before it getsprocessed by a collector. One advantage to this is that filtering ondata may be done earlier in the data stream at the device level.

According to another embodiment of the invention, the collectorframework may be easily modified to add event source connections tonetwork devices from a collector. This allows a group of event sources(e.g., network devices) to be collectively configured to use commonservices for configuration and health reporting via a single collector,without having to re-implement collector services for each event source.One advantage to using such a method enables an efficient way ofbuilding connectors into the framework.

Other advantages provided by this system may include the reducedresources required to deploy, manage, and monitor multiple deviceconnections. The collector framework may have greater control overconnections it deploys. Other advantages may exist.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a collector framework for implementing event sourcemanagement, according to an embodiment of the invention.

FIG. 2 illustrates a flow diagram for a method associated with thesystem of FIG. 1, according to one embodiment.

FIG. 3 illustrates a flow diagram for actively collecting data fromsources, according to an embodiment of the invention.

FIG. 4 illustrates a flow diagram for passive collection of data fromsources, according to an embodiment of the invention.

FIG. 5 a illustrates a health monitor user interface using a graphicalview, according to one embodiment of the invention.

FIG. 5 b illustrates a health monitor user interface using a tabularview, according to one embodiment of the invention.

FIGS. 6 a-6 c illustrate a series of exemplary dialog windows for addingand/or configuring a collector, according to one embodiment of theinvention.

FIGS. 7 a-7 b illustrate exemplary windows for adding and/or configuringan event source, according to one embodiment of the invention.

FIG. 8 illustrates an exemplary window for selecting filters, accordingto one embodiment of the invention.

FIG. 9 illustrates an exemplary window for a collector script viewer,according to one embodiment of the invention.

FIG. 10 illustrates an exemplary collector details window, according toone embodiment of the invention.

FIG. 11 illustrates an exemplary file chooser dialog, according to oneembodiment of the invention.

DETAILED DESCRIPTION

The present invention relates to a system and method for, among otherthings, continuously monitoring and collecting status information andevent data from one or more network devices on a computer network,parsing event data into normalized data structure, configuring one ormore collectors, filtering the event data, normalizing the statusinformation and event data, and displaying the normalized statusinformation and event data to a user (e.g., network administrator).

In one or more embodiments a collector framework system 1 (FIG. 1) mayinclude, a central manager 2, a central repository 4, a proxy manager12, a collector manager 14, collector directory 16, an event listener18, an agent engine 20, a collector 22, and/or one or more devices (26a-c). The components may be linked to communicate with each other inorder to enable the collection and management of status information andevent source data from one or more network devices. One or more of thecomponents may be implemented on the same or separate computer systems.

FIG. 1 illustrates interaction between the central manager 2, collectormanager 14 and agent engine 20, among other components of the system.The central manager 2 may function as the control unit for the collectorframework system 1. A central repository 4 may be located at orassociated with the central manager 2. Event data and status informationmay be received and collected at the central repository 4. The centralmanager also provides user interface views including a health/statusmonitoring interface 6, a collector configuration interface 8, and/or acentral repository interface 10. Each interface enables a user and/ornetwork administrator to visualize and control respective aspects of thecollector framework system 1. For example, the user may configure acollector (26 a) using the collector configuration interface 8. Theconfiguration information (e.g., collector scripts, configurationsettings, test data, etc.) may then be propagated from the centralmanager 2 out to a collector manager 14.

The collector manager 14 may execute several functions including, amongother things, connecting to an agent engine 20, creating a collectorprocess on the agent engine 20, managing listener ports for event datafrom the agent engine 20, processing event data from the agent engine20, sending processed event data back to the central repository 4,and/or sending status of the running collector(s) 22 to a proxy manager12. The collector manager 14 may start a collector 22 at the agentengine 20 by downloading a collector script from the central repositoryand/or receiving a collector data object. The downloaded script may bewritten to the agent engine to begin a collector process. For example, acollector 22 may be created based at least on the collector script 22.The collector data object may be retained at the collection manager 14in order to maintain a copy of one or more collector instances.

The proxy manager 12 may execute several functions including, amongother things, creating and managing connections to network devices 26a-c, which are the source of data for a collector 22, updating collectorrelated configurations, triggering the collector manger 14 to start aport and passing it the related collector data object, and sendingstatus (health) of collector(s) to the central repository.Communications with the proxy manager 12 are carried out in proxy mode.The agent engine 20 may connect to the proxy manager 12 in order toobtain information regarding which devices the collector should connectto.

The agent engine 20 may operate as a security device for event exchange.The agent engine 20 may instantiate a collector 22 based at least inpart on data obtained from the proxy manager 12, collector manager 14,and/or central manager 2. The agent engine 20 receives and manages theevent data from the collector 22 and sends the event data to thecollector manager 14 and/or status information about the collector(s) tothe proxy manager 12.

The collector 22 may be linked to one or more network devices 26 a-c andmay include a port configuration file 24 and a collector script 26.Collectively, the port configuration file 24 and collector script 26 cancreate a collector 22. Each may be configured independently of eachother by the central manager 2 or other entity. A unique portconfiguration file 24 is written for each collector. A portconfiguration file 24 specifies locations for a collector script.

The collector 22 may be deployed in order to perform in-depth monitoringof event data and state information from the network devices 26 a-c.Collector scripts may device specific. The collector manager 14 maydownload the appropriate collector script(s) from the central repository4 in order to run a collector 22 on one or more network devices. Thecollector script 26 parses the data from the network devices into anormalized data structure. This allows the system to effectivelynormalize event data from disparate network devices for easierstandardized reporting of event data. The flexible collector frameworkalso allows the collectors to be added, deleted, and/or modified to thesystem. By way of example, collectors gather data from source devicesvia many connection methods including, but not necessarily limited toSyslog, ODBC, JDBC, OPSEC, SSL, HTTP, and/or HTTPS.

One or more network devices 26 a-c may be grouped to create a networkdevice group (not shown) that feeds event data to a single collector 22.Network devices may be firewalls, routers, biometric devices,mainframes, databases and applications, among other things. Event datamay include security events, compliance events, and network events, etc.

The event listener 18 is a socket server on the collector manager 14that accepts connections from agent engine 20. The agent engine 20 sendsevent data to the event listener 18 for processing by the collectormanager 14.

A proxy connector 30 is a component that connects the agent engine 20 tothe proxy manager 12. The proxy connector 30 enables data and messagesto be communicated back and forth between the agent engine 20 and theproxy manager 12 during a proxy mode. The proxy connection 30 may beused to communicate status/health information from the collector 22 anddevices 26 to the central repository 4.

The collector framework system 1 provides various components that allowthe user to view network device level status and event data using one ormore collectors. The system may be using to perform various methodsassociated with collecting, retrieving, parsing, normalizing, analyzing,and reporting the status and event data.

FIG. 2 discloses a flow diagram for a method associated withinitializing the collection of event data, according to one embodiment.In operations the system components may be implemented to carry out themethod for initializing the collections of event data. A collectormanager process (e.g., collector manager 14) may be started, and then aproxy manager process (e.g., proxy manager 12) may be initialized(operations 302). When a proxy manager process starts up for the firsttime, it may register itself (e.g., with the central repository 4).Registering the proxy manager process may cause a proxy manager dataobject to be created. The ID of the proxy manager data object may bepersistent on the proxy manager machine (e.g., in a file) so that whenthe proxy manager process is started, it can identify itself to thecentral manager 2. After registration, the proxy manager process willthen be visible via the configuration interface 8 for adding collectorsand devices, discussed in further detail below. After the proxy managerprocess and collector manager process are started the event listeningprocess associated with collector manager 18 may also be initialized(operation 306). The collector manager process, proxy manager processand event listening process may be started and initialized in any orderand are not necessarily restricted to the order shown in FIG. 2.

The collector manager process may receive a collector script from aconnection to a central managing process in order to start a newcollector process (e.g., collector 22) (operation 304). The collectormanager process may write the collector script(s) to a collectordirectory 16, write the port configuration to a file, and spawn an agentengine process, pointing the agent engine process to read the portconfiguration which points to the location of the collector script inthe directory (operation 306).

The agent engine process may use a connection to the registered proxymanager process in order to determine which network devices to connectthe newly instantiated collector process to in operation 308. Forexample, since collector scripts may be device type specific, the proxymanager process can identify the network devices which may be connectedto the new collector process. As such, multiple devices may beconfigured for a single collector to form a device group.

As the agent engine process receives event data from the collector(s)(and network devices) it sends the events through an event communicationchannel to the collector manager process (operation 310). The collectormanager process can perform processing on the received events (e.g.,analysis, reporting, etc.) and eventually send the event data to thecentral repository 4.

Turning now to the actual method of data collection, FIG. 3 and FIG. 4are used to illustrate at least two collection models. The agent engineprocess may receive data from the one or more devices (e.g., devicegroup 26 a-c) in one of several ways. The collector process may eitherproactively retrieve data from the network device(s) or passively listenfor data from network device(s). Both provide device level visibility toa central manager 2.

In the proactive model 400, (see FIG. 3) the collector process maymanage a device queue 402 of one or more connections to each respectivenetwork device (26 a-c). The device queue 402 may cycle through eachconnection for each respective device through the connection thread pool404. After one chunk of data is retrieved from a connection to a device(e.g., connection #1 to 26 a), the connection is put back on an inactivequeue in the device queue. The raw data may be placed in a preliminaryoutput queue of raw data chunks for each respective connection. The rawdata transformer pool 408 may transform (e.g., filter, normalize, etc.)the raw data from the respective connections at the device level andrelatively early in the data stream. The process in operation 408 mayfilter the raw data to only include specific event data that the user isinterested in. Furthermore, the raw data may be normalized to a commondata structure specified by the over all system. This effectivelyenables a number of collectors which collect event data from disparatenetwork devices to be able to report data in a common data structurewhich may be more easily processed and viewed by a networkadministrator.

Thus, the filtered and normalized event data may be put on an outputqueue 410 in the order it was taken off of the preliminary output queue406 so that the order in which the data was initially received may bemaintained. The output queue 410 can maintain the output data for eachrespective connection and send it to the agent engine process onrequest.

Depending on the connector, the connector may have one connection perdevice group and therefore one thread in the pool 404 processing theconnection. In this case, the raw data transformer thread pool 408 wouldhave one thread as well. Further, the preliminary output queue 406 andthe raw data transformer thread pool 408 may be removed as they mat notprovide further functionality in the case of a single thread.

FIG. 4 illustrates a second model 500 wherein one or more collectorspassively listen for connections from data sources on respective localports. This model describes a plurality collectors and device groups.The collectors can listen on one and/or two ports (e.g., TCP and UDPport). Data sources will connect via one of these ports. In order todirect a subset of the incoming messages to a particular collector,there may be a filter that describes what attributes incoming datashould have in order to be sent to a particular collector. Part of thefilter can be the IP of the source machine sending the data, which willlimit the data sent to an agent to be from one source. However, the IPis optional in the filter, allowing the collector to be sent messagefrom all sources.

FIG. 4 is a diagram of how passive model may operate. Connections frommultiple data sources may be observed on one of several ports (502 a-c).The raw data may be collected in a preliminary output queue (504) of rawdata chunks by connection. The filtering process (506) may decide whichdevice groups should get the message put on their queue based on theirfilter. As a result, the raw data is filtered by device group intooutput queues (508 a-c) for the respective device groups. Other kinds offiltering besides device group level filtering are available. Forexample, filtering may be enabled at the connection level and/or thedevice level. When filtering at the connection level, some devices mayhave a filtering capability that is part of the protocol for gettingdata from the device. For example, a filter can specify the event datatypes to receive. When filtering at the device level, even if theconnected device does not natively support filtering, the connection maybe configured using a regular expression that will be executed againsteach record (the raw data) coming from the device. The filtered data canbe sent to the agent engine on request.

Alternatively, or in addition to the passive and proactive collectormodels, another method may choose to implement a Java database (DB)connectivity collector. The DB connector can support a connection to asingle database per device group. This implementation may prevent theagent engine 20 from having to track multiple databases and more thanone type of query to multiple databases.

The DB connector receives input from the agent engine 20 in order to getdata from the event source (e.g., device). This input is in the form ofan SQL query. The SQL query can be passed to the DB connector from theagent engine 20 using the transmit (Tx) state. The Tx state will writeout a value (the SQL Query), which can be forwarded to the DB connector.When the DB connector receives the SQL Query, it can execute the commandand put the result of the query on an output buffer. As the agent engineloops over its' receive (Rx) state, the output from the query will bepassed to the agent engine, a delimited chunk at a time. The DBconnector can set a variable in the chunks of data indicating if thechunk is the last piece of data returned by the last executed SQL query.When the appropriate variable is set indicating the end of data from thelast SQL Query, the agent engine may execute another Tx state to passthe next SQL query to the DB connector. In sum, the agent engine 20starts, the Tx state writes out the query, the query is forwarded to DBconnector (through the flow/slave mode/connection manager), the agentloops over its Rx state, receives output of SQL query one line at atime, and when there is no more data from the SQL Query, the agent canhit another Tx state to send the next SQL Query, which the DB connectorcan get and execute.

All or some collectors including active, passive and DB, report on theirhealth also called status. Health statistics can be sent to a healthviewer user interface 6 for display to a user. Each collector connectioncan have two points that can report health independently. They are thedata ingress point and the data egress point.

In the data ingress point the health will be reported at the point thatthe collector connection gets data from the data source. For example,this may include the status of how much data is being received from asingle connection or if the connection is connected or not. There can be1 to n number of data ingress points per collector connection. Thenumber of data ingress points does not necessarily equal the number ofdata egress points. The kind of health information that can be collectedat the data ingress point depends on the type of collector connectionused. The units of the data coming in can be either bytes or recordsdepending on whether the collector knows how to break up the incomingdata at the ingress point into records. If it does not, then the unitsmay be bytes. There may be at least three types of connections for whichdata ingress health information is reported. They may be connectionoriented server, connectionless server, and/or client connections.

Connection oriented network devices connections may include, forexample, Syslog TCP and HTTPS. Health information reported in thiscontext may be based on individual connections from remote data sourcesto the collector. Health information from the collector connection mayinclude status, status over time, and/or identifier information. Statusinformation may include whether the collector connection is successfullybound to a listening port and/or whether the collector connection isconnected to a device. If not connected an error state may be displayed.Eventually, if still not connected, then the health information may beremoved. Status over time may include, total data (bytes/records)received, total time connected, interval data (bytes/records) received,interval time, and/or time of last data received. Identifiers mayinclude local host/port identifier and sender's host/port identifier.

Connectionless device connections may include Syslog UDP and SNMP.Health information reported in this context is based on each uniquesender's IP address. The health report keeps track of the sender IPaddresses that are observed. This is because there is no connectionsession to tie the health to. Similar to the health reports for aconnection oriented server, the health information for a connection lessserver may include status, status over time and/or identifier.

The third type of collector connection for which data ingress healthinformation may be reported is in client connections. Health informationreported in this context may be based on connection to each device(which could be a log file, process, or socket). Health information mayinclude status, status over time, and/or identifier information. Statusmay include connection status to source. Status over time may includetotal data (bytes/records) received, total time connected, interval data(bytes/records) received, interval time, time of last data received,and/or time of last data requested. Identifier information may be thename given by user to the client connection and/or connector specifiedidentifier for what is connected to (e.g., filename, process commandline, host/port, etc.).

The second point in the collector connector that may be monitored forhealth information is the data egress point. Health may be reported atthe point that the collector connection passes the data out to the agentengine. There may be one data egress point for each device group. Thenumber of data egress points will also equal the number of collector 22processes. Health information reported at the data egress point may bethat of the device group. The health information collected at this pointmay include status, status over time, and/or identifier. Status overtime may include number of records in queue, total number of recordssent out from the device group, total time that this device group wasactive (equals total agent engine running time), number of records sentout from this device group in last interval and/or the time of lastinterval. Identifier may include agent (port) name, proxy name, and inthe case of server type connectors, this may be the name given by userto the filter that is associated with this egress point.

There may be at least two scenarios when health can be reported. Thecollector manager 14 or other entity may periodically poll thecollectors for health status information. This may be synchronous healthreporting. In another scenario the connector can asynchronously reporterror status to the collector manager 14 or other entity, immediatelywhen the error occurs.

User Interfaces:

The system of the invention provides graphical interface capabilitywhich enables the user to configure collectors, monitor event sources,and/or manage collectors, among other things. The system may provide atleast a health view interface 6, configuration interface 8, and/orcollector repository interface 10. The health view interface 6 may,among other things, display one or more application windows that providethe status of objects in the current configuration. The configurationinterface 8 may, among other things, display one or more windows forviewing and editing current configurations for collectors, eventsources, filters, and/or testing. The central repository interface 10may, among other things, display running collector repositories andcollector details, and enable collector importing. One or more of theinterfaces may be combined into one or more window displays.

Health View Interface:

The health view interface 6 can display the current status of entitiesin the system including the proxy manager 12, collector manager 14,collector 22, device group and/or individual device(s) (26 a-c). Theentities may be displayed as nodes in a hierarchical graphical format,with the nodes connected by arrows (or other visual means) indicatingthe direction of the flow of data. The icons used for each node cansymbolically represent the type of entity the node represents. The iconof the node can change depending on the state of the entity the node isrepresenting (e.g., running/stopped). FIG. 5 a provides an examplerepresentation of the health view interface 6 in the graphical format.Optionally, the health view may be displayed in a tabular format asshown in FIG. 5 b. Both formats provide device level visibility forstatus and connection information.

The tabular view 70 provides at least two status views. They areconfigured status 72 and actual status 74. The configured status may bethe on/off state for which the node is configured to be in. The actualstatus 74 of the node may be reported by the actual running collectormanager. The connection information column 76 may provide a textualdescription of the event source connection. The error column 78 may be atextual description of an error that occurred in the running node.

In the graphical view shown FIG. 5 a, the graph arrows (or otherindicator) not only show how the nodes relate to each other, but alsothe state of the data flowing between the entities. The color or textureof the arrow can show whether the connection is alive or dead (e.g., isan event source turned on such that it could possible send data to acollector). A tag on the arrow (not shown in the figure) can display thecurrent records per second flowing from one node to another. Thegraphical view may also display a tool tip when a mouse (or otherselecting device) “hovers” over a node. The tool tip can display asnapshot of more detailed status information. The graphical view alsoprovides the option for selecting from various graphical layouts from apull down menu 64 or other selection means. The overview portion 66 ofthe graphical view allows the user to quickly move across the graphicalview. This is particularly useful when there are many entities in thescreen and the user is zoomed in such that all of the entities are notviewable at the same time.

In the graphical view each node can have a number of right-click actionsthat can be taken on the node. It should be understood that otherselection means besides right clicking may be used to result in the sameactions. The right-click actions available when clicking on a node canbe context sensitive to the type of node being clicked on. For examplethe right-click actions that may be supported for a proxy manager 12 mayinclude proxy status details, configure proxy (opens the configurationdialog for the selected proxy), proxy events (opens an active view thatdisplays events coming from the selected proxy), start proxy, stopproxy, and/or add collector (opens an “Add Collector” screen that canguide the user through the process of adding a collector to the selectedproxy).

Right click actions and/or options associated with the collector 22 mayinclude collector status details, collector data tap, collectorconfiguration, collector events, add event source, start collector,and/or stop collector. The collector data tap can open a window thatshows the user the live stream of raw data that is currently coming fromthe event source or flowing through the selected collector. The raw datamay be viewed as filtered and/or normalized according to the collectorfunctions. The collector configuration option can open a configurationdialog for the selected collector. Aspects of collector configurationare described in further detail below. The collector event option opensan active view that displays events coming from the selected collector.The add event source option can open a window that guides the userthrough the process of adding an event source to the selected collector.

Right click actions and options associated with a network device mayinclude device status details, device data tap, device configuration,device events, start device, stop device and/or created correlation rulefor device. The device data tap option can open a window that willdisplay raw data (data not processed by the collector script) comingfrom the data source the device is connected to. The deviceconfiguration option opens a configuration dialog for the selecteddevice. The device events option can open an active view that displaysonly events coming from the selected device.

Other actions and options that may be associated with both thecollectors and devices may include among other things, remove, open rawdata tap, move, and/or clone. The remove option can delete a selectedevent source and/or collector from the system. The open raw data tapopens a window that shows the user the live stream of raw data that iscurrently coming from the event source or flowing through the selectedcollector. The move option allows the user to move the selected entityfrom its current parent entity to another parent entity.

For example, in the case of event source, it allows the user to move theselected event source off of the collector that it is currentlyconnected to and onto another collector. The clone option allows theuser to create a new entity that has its configuration informationpre-populated with the settings of the currently selected entity. Forexample, in the case of event source, it allows the user to create a newevent source that is pre-populated with the configuration setting of thecurrently selected event source. This allows a user to quickly create alarge number of similar event sources without having to retype in thesame information over and over again.

Actions that may be common to all entities may include status details,configure, open active view, start, stop, tabular/graphical view, and/orzoom. Status details can display all information known about the statusof the selected node. Status information is the runtime status of thenode (e.g., On/Off, EPS, errors, etc.). A status details dialog displaysstatus details of the selected node as well as all parent nodes (forquick viewing, since the status of parent objects may affect childnode).

The configure option allows the user to edit the configurationinformation of an entity. The open active view option opens a fullfeatured active view window that displays events that have beengenerated by data from or flowing through the selected entity. The startoption sets the entity to be running by setting its state to ON.

The stop option sets the entity to be stopped by setting it's configuredON state to be false. The “show in tabular/graphical view” optionswitches to the other view (to tabular view if on graphical view, or tographical view if on tabular view) and automatically selects the entitythat is selected in the current view. When switching to graphical view,it also zooms in on the selected entity. The zoom option (in GraphicalView) can be used to zoom in on a selected object in the graphical viewdisplay.

The health view interface described above may be used in an onlinehealth mode. In addition to the “online” mode there is also a designmode. The design mode may be represented as panel within the health viewinterface. This panel may allow the user to configure collectors andevent sources (devices) without actually tying them to an instance of areal collector manager. This enables the user to view and edit nodes inthe design mode while displaying no status information about these nodesbecause they are not actually tied to a collector manager. After gettingthe configuration of the collector and event source nodes just asdesired in the design mode, the user can “deploy” the configuration tothe collector manager, which can move the collector and event sourcenodes from the design mode to the online mode.

Configuration Interface:

The health viewer interface 6 can provide access to configurationinterfaces 8 for selected nodes. The configuration interface 8 allows auser to edit and view current configurations. Similar to the health viewinterface 6, the configuration interface 8 may display windows in eithera tabular and/or graphical view. The configuration interface 8 allowsthe user to configure various aspects of the system including addingcollectors, adding event sources (e.g., device and/or device group(s)),raw data filtering, and/or device testing.

FIGS. 6 a-6 c illustrate a series of exemplary windows for adding and/orconfiguring a collector. In FIG. 6 a the user can choose which device togather data from. The list of devices 82 to choose from will depend onthe set of collector scripts in the collector repository. Sincecollector scripts may be device specific, the user can choose to gatherdata from a device for which they have a collector that knows how toparse data from the device. This list 82 may appear as a table withthree columns including vendor 84 (the name of the vendor that makes thedevice), product 86 (the product name of the device), and/or version 88(the product version of the device).

If the user does not find the device they are looking for in the tablebut they do have the collector package (e.g., collector script) for thedevice they are looking for, there may be a button on this panel thatlaunches the collector repository import interface to allow the user toimport a new collector parser into the repository. The collectorrepository interface is discussed in further detail below. After theuser is done with the collector repository import interface, they can beautomatically returned to the add collector panel and the table in panelwill have been updated to reflect the additional collector added to thelist.

FIG. 6 b illustrates an exemplary window 90 for selecting a collectorscript (parser). The chosen collector script may be used to process datafrom the device they selected in the first screen shown in FIG. 6 a. Theuser may have the option to choose from a list of compatible collectorscripts that specifically state in their meta-data that they can processdata from the selected device. The user may also select via selectionmeans 92 to view all available collector scripts. The list of collectorscripts 94 to choose from can appear in a table with the columns below.When a user selects a collector script, the details from the meta-datafrom that collector script 96 can be displayed below the list ofparsers. The list of collector scripts may be sorted by column includinga column for compatibility 94 a, parser name 94 b, author 94 c, version94 d, device vendor 94 e, device name 94 f, and device version 94 g. Thecompatibility column 94 a may be expressed using an icon indicative ofcompatibility. This column should have the compatibility icon in it whenthe collector script is known to be compatible with the device. Theparser name column 94 b stores the collector script name. The authorcolumn 94 c indicates the organization (e.g., engineering) or individualwho created the script. The version column 94 d stores the version ofthe script (not the version of the device). The device vendor 94 e showsthe name of the vendor of the device that the script can process datafrom. The device name 94 f shows the product name of the device that thescript can process data from. The device version 94 g is the productversion of the device that the script can process data from. All columnsshould be sortable.

If the user does not find the script they are looking for in the tablebut they do have the collector package for the script they are lookingfor, a Collector Repository Import Wizard may be launched to allow theuser to import a new Collector script into the repository. After theuser is done with the Collector Repository Import Wizard, they willautomatically returned to this window and the table in the window willhave been updated to reflect the additional Collector Parser they'veadded to the repository.

FIG. 6 c illustrates an exemplary window for configuring the addedcollector. The user can specify the name of the collector in text box102, set the collector parameters, choose whether the collector shouldbe running, and set the filter on the raw data flowing into thecollector. Also, there may be a button 104 on this window that allowsthe user to save their changes to the collector and immediately jump tothe “Add Event Source” wizard to begin adding Event Sources. Configuringan existing collector will look similar to the window shown in FIG. 6 c.

FIGS. 7 a and 7 b illustrate two windows that allow the user to addand/or configure an event source (e.g., device and/or device group(s)).In FIG. 7 a the user may select a connection method. The user can selectwhich connection method to use in order to get data from the device (thedevice was chosen when adding the collector). The list of connectionmethods to choose from may depend on the connections supported by thecollector manager 14. The user can choose a connection method that isimplemented by the collector manager 14. The user may have the option tochoose from a list of compatible connection methods, that is, connectionmethods that are specifically stated in the associated collectormeta-data as compatible with the collector or all connection methods.The list of connection methods to choose from can appear in a table withcolumns for compatibility 124 and/or name 126. Compatibility may beindicated by an icon when the connection method is known to becompatible with the associated collector. The name column contains thename of the connection method.

FIG. 7 b illustrates a window 130 that may be used to configure theevent source. The user can specify the name of the event source 132, setthe connection method configuration properties, set the offset in theevent source data 134, choose whether the event source should berunning, set the filter on the raw data flowing through the event source136, and test the connection 138. Also, there may be a button thatallows the user to save changes 140 to the event source and immediatelystart adding another event source by jumping to the beginning of the“Add Event Source” windows. After clicking finish, if the user selectedto have the event source running and the associated collector is alsoset to be running, the dialog should appear to tell the user if theevent source and collector did indeed startup successfully. Configuringan existing event source may look similar to the window shown in FIG. 7b.

The configuration interface also allows the user to test theconfiguration of a device. This allows the user to establish with morecertainty that the configuration parameters they entered into theconfiguration interface are correct. Within the device configurationdialog (FIG. 7 b), there may be a selection means 138 that, whenselected, will test the device configuration. Devices that have beenalready saved to the database as well as devices that have not beensaved to the database can be testable. A user can edit the configurationof a device while the device is in the running state.

Furthermore, a user can test a device that is in the running state. Thetesting of the device can be done independently of the already runningdevice, therefore the fact that the user is testing the device may notaffect the data flowing into the running device. The device may beaffected when the user saves the configuration change. When the testbutton is clicked, a new window may pop up. The new window may havedisplay components including: a display for the values of the attributesof the Proxy Device Status object from the Device being tested; adisplay for the data from the Device (the data will stream from theDevice to the GUI), and/or a stop button that the user can click to stopthe testing of the Device. Closing the window may also implicitly stopthe testing of the Device. Device testing may also include a way oftesting the Device and Device Group filters.

The event source and collector may be further configured to filter theraw data collected there from. FIG. 8 illustrates a filtering window 150with a table of filters 152. Each row (152 a-e) in the table is a singlecriterion for matching against the incoming data. Together, all of therows 152 a-e make up the overall filter criteria. The filter criteriaare matched from top down, with the first match being the one performed.Once the first matching criterion is found, the criteria under it arenot checked. The filter manipulation buttons 154 a-d enable the user toAdd, Remove, and reorder the criteria in the table. The default actionselection 156 allows the user to specify whether to include or excludethe data if the data does not match any of the criteria specified in thetable.

The columns of the table may include column name 152 a, condition 152 b,value 152 c, action 152 d, and/or enable 152 e. The incoming data may bea map of key-value pairs. The column name 152 a is the name of the keywhose corresponding value this row's criteria will be matched against.The condition column 152 b allows the user to select the kind ofcriteria that will be used. Regular expressions may be supported. Theremay also be other kinds of criteria types typically available forStrings. The value column 152 c may indicate the actual criteria tomatch against. This value will be used differently based on thecondition selected. The “value” column can be automatically validated bythe interface for acceptability. If there is an error in the valuecolumn (such as empty value or invalid regex) that cell of the tablewill be highlighted and a description of the error will be displayed inthe lower left of the dialog. The column for action 152 d specifies anaction to be taken if there is a match. Actions may be “include” (whichwill result in data passing the filter) and “exclude” (which will resultin the data being dropped by the filter). The enable column 152 e allowsthe user to set a flag. If this flag is not checked, then the row willbe skipped, just as if it was not part of the filter. This allows a userto easily disable part of a filter temporarily.

Collector Repository Interface:

The collector repository interface 10 enables users to update acollector in one location and have the change automatically propagateout to the collector manager that is controlling the collectors. Theremay be at least three collector repository screens that make up theinterface functionality. They may include a collector repository viewer,collector details, and/or import collector wizard.

FIG. 9 illustrates an exemplary collector repository viewer. Thefunctionality provided by this window allows the user to view thedetailed information about the collector (by double clicking (or otheraction) on the collector from the list). FIG. 10 illustrates anexemplary collector details window which displays information from thecollector's meta-data and instances of the collector currently deployed.

The collector import wizard allows the user to import collector scripts.FIG. 11 illustrates a first screen in wizard that provides the user witha file chooser dialog for browsing stored collector packages. The filechooser may have a built-in filter to show files with the valid fileextension for a collector package. Various outcomes may be performedbased on the selected collector package. If the ID of the collectorbeing imported does not match any of the collectors currently in therepository, then the meta-data of the collector being imported will bedisplayed and the user can finish and proceed to import the collector.They also have the option of clicking “Finish and Deploy”, which willimport the collector package and after importing is successful it willbring up the “Add Collector” wizard.

If the ID of the collector being imported does match one of thecollectors currently in the repository, then the meta-data of theexisting collector may be displayed as well as the meta-data of the newcollector being imported. The user may be asked to confirm thereplacement of the collector currently in the repository with thecollector selected for import. In this case there will also be aselection means that allows the user to select whether to update at nextrestart or update immediately.

If update at restart is selected, the updates in the newly importedcollector will not be pushed out to any deployed collectors until thenext time the collector restarts (when collectors restart the collectorManager checks against the repository to make sure it has the latestversion of the collector). If the update immediately option is selectedthis forces all currently deployed collectors to restart, causing themto immediately download from the collector repository the latest versionof the collector being imported.

After the user confirms import of the collector script, the collectorscript will be uploaded to central repository. It may be saved to theserver in a new filename. After being uploaded, an agent script dataobject for this collector script will either be created and saved to theDB (if this is a new collector script) or the existing agent script dataobject will be updated (if this is an update of the collector script).One of the attributes in the agent script data object will be the hashof the zip file of the new collector script.

When a collector Manager is starting a collector, it may first checkthat the hash of the collector script it has matches the hash of thecollector script in the central repository. If they do not match, thenthe collector manager will download the collector script from thecentral repository, replacing the collector script it has locally. Itwill then run this collector script.

If replacing an existing script already being used, a window may pop upallowing the user to select which agent ports (port configs) they wantto push this updated script out to. The list of agent ports displayedmay be in two categories 1) agent ports running the same agent for thesame version of the same device (e.g., the case of Engineering bug fixor enhancement) 2) agent ports running the same agent for a differentversion of the same device (e.g., the case when an agent is updated tosupport a newer version of the same device).

An agent status interface can display when an agent script is not insync with the server (because the Agent has been running since beforethe script was updated in the central repository). If the agent scriptis updated in the repository, port configurations will automatically beupdated (may require a port stop/start).

The foregoing presentation of the described embodiments is provided toenable any person skilled in the art to make or use the invention.Various modifications to these embodiments are possible, and the genericprinciples presented herein may be applied to other embodiments as well.For example, the invention may be implemented in part or in whole as ahard-wired circuit, as a circuit configuration fabricated into anapplication-specific integrated circuit, as a firmware program loadedinto non-volatile storage or a software program loaded from or into adata storage medium as machine-readable code, such code beinginstructions executable by an array of logic elements such as amicroprocessor or other digital signal processing unit, or may includeother implementations.

Embodiments of the invention include a computer program containing oneor more sequences of machine-readable instructions describing a methodas disclosed above, or a data storage medium (e.g., semiconductormemory, magnetic or optical disk) having such a computer program storedtherein. The invention is not intended to be limited to the embodimentsprovided above, but rather is to be accorded the widest scope consistentwith the principles and novel features disclosed in any fashion herein.The scope of the invention is to be determined solely by the appendedclaims.

What is claimed is:
 1. A collector framework system including at leastone collector for collecting event data from one or more networkdevices, the system comprising: a collector manager for storing acollector script; configuring means for configuring a collector based onthe collector script; a proxy manager for determining one or morenetwork devices to connect to the collector based on the collectorscript, wherein the collector collects event data from the one or morenetwork devices; and receiving means for receiving the collected eventdata from the collector manager.
 2. The system of claim 1, wherein theconfiguring means further includes one or more user interfaces enablinga user to view collected event data.
 3. The system of claim 1, whereinthe collector script is device specific.
 4. The system of claim 1,wherein the event data comprises security events, compliance events, ornetwork traffic events.
 5. A computer based method including at leastone collector for collecting event data from one or more networkdevices, comprising: configuring a collector process based on thecollector script; determining one or more network devices to connect tothe collector based on the collector script, wherein the collectorprocess collects event data from the one or more network devices; andreceiving the collected event data from a collector manager.
 6. Themethod of claim 5, wherein the configuring further includes one or moreuser interfaces enabling a user to view collected event data.
 7. Themethod of claim 5, wherein the collector script is device type specific.8. The method of claim 5, wherein the event data comprises securityevents, compliance events, or network traffic events.